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FOREWORD 



This research was completed under Project 7734, Development of Meihods for 
Describing, Evaluating, and Structuring Air Force Occupations; Task 7734Gi, 
Development and Appraisal of Methods for Job Evaluation. 

The present study compares the reliability of job ratings made by individuals to job 
ratings made by panels of judges. Results provide information useful to the evaluation of 
Air Force jobs. Certain findings may also be applicable to other situations where boards 
or panels are used to reach personnel decisions. 

Appreciation is expressed to MSgt R. E, Sprouse, Mr. C. L. Cannon, and Mr. T, C, 
Whitney who conducted the expei.mentaJ sessions, and to Mr, R. S. Massar for developing 
the compute; programs for the statistical analyses. Dr. R. E. Christal provided assistance 
in interpretation of results, and SMSgl D. K. Cowan handled the administrative and 
statistic^ details. 

This report has been reviewed and is approved. 

John G. Dailey, Colonel, USAF 
Commander 
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ABSTRACT 



This study investigated two approaches for obtaining job ratings in order to 
determine which procedure provided the most stable ratings, given a constant number of 
raters. Specifically, it compared reliability estimates deteimined by averaging across 
individually obtained job ratings and reliability estimates based on consensus ratings from 
interacting panels. To investigate reliability of job ratings obtained from individuals and 
from groups or panels of raters, 450 basic airman rated 100 brief job descriptions under 
three conditions; (d) individually, (6) in a 3-man panel, and (c) in a 5 -man panel. Analyses 
revealed that, estimates of mean reliability were larger for individually obtained ratings 
than for 3-man or 5-man panel ratings. There was also a trend for mean rating time to 
increase with an increase in panel size. Present findings tend to support the procedure of 
averagin'" across individual ratings, rather than the use of ratings from panels, in order to 
obtain t are stable results. In terms of reliability, lime required, and number of raters, the 
individual approach appears more economical and efilcioal than the board or panel 
procedure. ^ 
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SUMMARY 



Hoggalt, R.S. &. Haiel, 3.T. Reliability of individual versus group job pay AFHRL-TR-70- 10 Lackland 

AFB, Tex.: Personnel Rescardi Division, Air Force Human Resources Laboratory, April 1970. 

Problem 

The Air Force often uses boards to reach personnel decisions such as those made in promotion 
selection, school nomination, and job evaluation . In some board situations, averages of ratings obtained 
individually are used as a basi5 for decisions concerning persons or jobs. In contrast, consensus ratings of 
boards may be used to reach decisions. For either approach, stable ratings reflecting high agreement among 
judges are desirable. 

fn terms of job evaluation, this study investigated l^vo procedures to determine which provides the 
most stable ratings, given a constant number of raters. Specifically, reliabilily estimates based on ratings 
obtained individually were compared to reliabilily estimates based on consensus ratings oolained from 
interacting 3-man and 5-man panels. In addition to reliabilily, the economy and efficiency of the two 
methods A'ere considered. 

Approach 

To examine individual versus panel differences in reliabilily of job pay ratings, three experimental 
conditions were selected to produce (u) individual ratings, (F) ratings of 3-man panels, and (c) ratings of 
5-man panels. The task for subjects in each condition was to provide pay ratings of brief job descriptions 
using a 9-point scale. The job sample included 100 brief descriptions distributed over high to low 
socioeconomic levels, but each indi/dual or panel was required to rate only 20 jobs. The subjects were 450 
basic airmen, with 1 50 subjects randomly assigned to each of the three conditions. The number of raters in 
each condition was held constant, although the number of ratings per job vaiied as a consequence of the 
division of subjects into 3-man and 5-man groups. 

Results 

Inlraclass correlation analyses were used to con. pare the reliability of job ratings for the individual, 
3-man panel, and 5-man panel conditions. Results of these anaVses revealed that estimates of mean 
rePability were larger for the individual condition than for the two panel situations, when estimates 
accounted for number of ratings per job. This finding was consistent when projected across various si/e 
samples. Multiple range tests revealed a statistically significant difference between the mean time required 
for the individual and the S-man panel situations, and there was a trend for the time required to increase 
with an increase in panel size. Other analyses between individual and panel pay ratings found the 
distributions of the mean pay ratings for the 1 00 jobs Axre similar, and there were no significac.l differences 
among rating means for the three conditions. 

Conclusions 

When both reliability and time required to make ratings are considered, an individual rating situation 
appears preferable to a group rating situation. The evidence supports the procedure of averaging individual 
ratings, raiher than dividing raters into panels and obtaining consensus group ratings, in order to obtain 
more stable mean job ratings. In terms of reliability and economy of lime, averaging ratings across 
individuals appears more efficient. More time and more raters niay be required fir the panel situation to 
achieve reliability estimates comparable to those achieved by the individual situation. Further, (he number 
of nan-hours required by 5-man panels suggests that the use of larger sized panels may be prohibitive from 
a practical standpoint. 

This summary was prepared by J. 7. Ha?cl, Personnel Systems Branch, Personnel Research Division, 
Air Force Human Resources Laboratory, 
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RELIABILITY OF INDIVIDUAL VERSUS 
GROUP JOB PAY RvMlNGS 



I. INTRODUCTION 

The Air Force often uses boards or panels lo 
reich personnel decisions such as those made in 
promotion selections, school nominations, andjob 
evaluations. In some board situations, averages of 
ratings obtained individually are used as a basis for 
decisions concerning persons or jobs, In contrast, 
consensus ratings of boards may be used to make 
decisions. For either approach, stable ratings 
reflecting high agreement among judges are desir- 
able, in order to place greater reliance on the 
validity of decisions. 

A cursory review of the literature concerning 
small group behavior over the past fifty years 
suggests certain controversy persists concerning 
the comparative effectiveness of group perform- 
ance and individual performance. Part of this 
controversy may be due to insufficient distinction 
between reliability and validity in defining supe- 
rior performance. In terms of job evaluation 
reliability, this study made certain comparisons 
between individual and group rating procedures. 

In the evaluation of Aii Force jobs, a 
systematic procedure is followed v^iiercby the 
judgments of experienced officers are used lo 
delernvne equitable pay grades for various jobs. A 
senes of studies has been undertaken lo investigate 
variables and job requirement factors involved in 
this procedure. Some of the variables which have 
been investigated are number of raters per job, 
length of job descriptions, time spent on evalua- 
tion, and different categories of raters (Chrislal, 
Madden, & Harding, I960; Hazel, 1966; Hazel & 
Cowan, 1966). 

The present study compared the stability of job 
pay ratings obtained individually from judges w ith 
ratings obtained from different sized panels of 
judges. The panel ratings represented the con- 
sensus of interacting groups. Two panel rating 
situations were compared with an individual 
situation in order to determine which approach 
yielded the most stable ratings, and was most 
efficient with regaid to time required and numbe ■ 
of raters. More specifically, the question was 
examined whether, given a constant number of 
raters, more reliability can bt obtained by 



averaging across individual ratings or by obtaining 
consensus ratings from 3-man and 5-man panels. 



II. METHOD 

To examine reliability differences between in- 
dividual and group job pay ratings, three experi- 
mental conditions were selected. The task for 
subjects in each of these conditions was lo provide 
pay ratings of brief job descriptions, either in- 
dividually or as panel members. A total of 100 job 
descriptions was used, but each individual or panel 
was required lo rate only 20 jobs. The essential 
features of the three conditions, instructions, 
subjec's, job descriptions, and 'aling scale are 
described subsequently. 

Experimental Conditions 

In one condition, individual ratings w'ere 
obtained on 100 job descT'iptions, In »wo con- 
ditions, panel ratings w'cre obtained for the same 
job descriptions. The tasks and compositions of 
the three conditions were as follows: 

1. hdividual ratings. For this condition, each 
subject indiWdually rated a set of 20 job descrip- 
tions. There were five sets of descriptions, or a 
total of 100 jobs. Each job received 30 ratings, so 
the total ratings for this condition was 3,000 (i.c., 
100 jobs limes 30 rating:). 

2. J-tnan panel ratings. For this condition, each 
subject was randomly assigned lo a 3-man panel. 
Each par:cl icached a consensus, then taled a set of 
20 job de.^icriplions. As before, there were 100 jobs 
f5 sets), but crch job received only i^i 3-mau 
panel ratings. This required 50 panels and yielded 
a total of 1 ,u00 ratings (i.e., 10 panel ratings per 
i'jb limes 100 jobs). 

3. S man panel ratings. For this condition, each 
subject was randomly assigned to a 5-man panel. 
Each panel reached a consensus, then rated a set of 
20 job descriptions. As before, there were 100 jobs 
(5 sets), but each job received only six 5*man 
panel ratings. ' irty panels were required, and this 
yielded a total of 600 ratings (i.e ., 6 ratings per job 
limes 100 jobs). 
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Subjects 

The subjects used to obtain pay ratings on the 
100 job descriptions were 450 basic airmen 
assigned to Lackland Air Force Base, Texas. 
.‘Mthough the number of ratings per job differed 
for the three conditions {i.e., 30 individual ratings, 
10 ratings from 3-man panels, and 6 ratings from 
5 nran panels), the number of raters per job was 
held constant (i.e., 30), Consequently, 150 
subjects were randomly assigned to each of the 
three conditions. 

iProcedure 

Williin piCscribed limits, the procedures and in- 
structions wcic sirriilar for the three conditions. 
Lssontially, each subject in condition 1 or each 
panel in conditions 2 and 3 was provided a set of 
20 job descriptions and asked to rate these jobs on 
a 9-point scale. All individuals (condition 1) and 
all panels (conditions 2 and 3) made their ratings 
sinuittancously; however, the three groups were 
S'.parated The time in niinutc^ was recorded f'^r 
individual and panel ratings. Detailed instructions 
for the three conditions are given in Appendix i. 

Job Descriptions 

Foul requirements were considered in the 
selection of the 100 job description sample: 

1. Suitability of job descriptions for use with 
the rater groups (i e., basic airmen). 

2. Availability of selected job descriptions in a 
standard reference s.jurce. 

3. Reduction of context effects by inclusion of 
hig)i-, moderate-, and low-pay jobs in the job 
sample. 

4. Identification of some basis for an empirical 
determination of the pay spread among jobs. This 
requirement appeared necessary to insure adequate 
variation among jobs, and pro'/ided a realistic 
fra.nework for determination of a monthly pay 
rating scale. 

The first requirement, suitability, was rcco" 
ni/ed by selecting commonly encountered c*r 
familiar jobs. Unique titles or higlily specialised 
jobs (c.g., agronomist, flumcr, zanjero) wtic 
avoided. Rclcvar^cy of certain job descriptions to 
Air Force jobs was also considered (c.g., airplane 
mechanic, pilot). Availability of job descriptions in 
a standard source was resolved by selection of jobs 
listed in the DiefiotJary of Occuj^atioval 7 Vt/«’ 5 , 
Volume I (196$). 
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A study by Madden (1960) demoitstrated that 
inclusion of only liiglwalue (or only low -value) 
jobs together introduced serious bias in job evalua- 
tion scores. To help reduce such context effecis 
and also to form some basis tor an empirical pay 
spread among jobs, the occupations ar:d social 
status report by Reiss (1961) was used extensively. 

The problem outlined by Reiss was the investi- 
gation of the prestige or social status of various 
occupations, considering such variables as 
education and income levels. Socioeconomic in* 
dexes were obtained on 425 jobs outlined in the 
occupational classification of the 1950 United 
States Census of Population. These indcx^'S were 
constructed on a population of maU.> in ihe 1950 
experienced labor force. The 4?.S |ob^ were 
grouped into )2 major occupalion.il cjiegorics 
(Reiss, 1961, pp. 154-155). Titles id these major 
occupational gioups are given in Appendix II. 

To counterbalance context effects, job samples 
from various major occupational groups at differ- 
ent socioeconomic index levels were selected 
(Reiss, 1961, Tabic B-l, pp. 264-275). The distri- 
bution of the 100 job samples for 12 occupational 
groups at 10 socioeconomic index -evels is given in 
Table 1 . Tlie median number of jobs selected from 
the sociocconom'c index levels was approximately 
10, with the range from 19 to 5. jobs for tw'o 
groups, private household -nd unreported occupa- 
tions, were omitted. 

The basis used to determine an empirical pay 
spread among jobs was the 1^66 median earnings 
of male civilians in the same 10 occupaiional 
groups previously described. This information v as 
obtained from the Bureau of the Census publica- 
tion Sfcjfii'ficd/ Ahstmcl of the United States 
(1968). The annual and monthly incomes of the 
to occupational groups are given in Appendix [1. 
As shown, there was considerable variation in 
monthly pay of the occupational groups. This 
variation, ranging from S767 to S21 5 was assun.^d 
adequate to reduce spurious reliability values. 

After determining the number of jobs desired in 
the Various socioeconomic level and occupational 
group categories, the actual selection of a repre- 
sentative sample of 100 brief job descriptions was 
undertaken. This action consisted of reviewing job 
descriptions in the DicrioMdry of Occupa*iothd 
TitS's (DOT), and evaluating jobs m terms of th.e 
mur requirements, suitability, availability, c, *^text 
ivffect, and empirical pay spread. After 100 jobs 
Were selected, abstracts of tliC DOT descriptions 

U 
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Tabfe 1. Distribution of 100 Job Description Sample by Socioeconomic 
Index and Occupational Group 



Occupatronii 

Group* 








socioeconomic index 










0-9 


10-19 


20-29 


30-99 


40-49 


50-S9 


60-69 


70-79 


80-89 


90-99 


Total 


1. Professional & Technical 










4 


1 


3 


7 


9 


7 


31 


2. Farmers & Farm Managers 












1 










1 


3. Managers, Officials & 
























Proprietors 








1 


1 


2 


2 


2 


2 




10 


4. Gericol 






2 




7 


2 


2 








13 


5. Sales 








1 














1 


6. Craftsmen & Foremen 




4 


4 


3 


6 


1 


2 








20 


7. Operatives 


1 




3 


4 


1 


1 










12 


8. Private Household 






















0 


9. Service (except household) 


2 


5 




2 














9 


1 0. Farm Laborers & Foremen 


1 




















1 


1 L Laborers (except farm/mine) 


1 


1 


















2 


12. Occupations not reported 






















0 


Total 


5 


12 


9 


11 


19 


8 


9 


9 


11 


7 


100 



“After Reiss (1961). 



were written and edited. The objective for these 
abstracts was brevity, avoidance of technical 
terms, and focus on the most prominent duties of 
a job. The 100 jobs were randomly sorted info one 
cf five sets of 20 job descriptions. The lists of jobs 
as they were grouped on the five sets of rating 
forms are given in Appendix 111. 

Rating Scale 

A 9-poi: t monthly pry rating scale was derived 
from the previously described empiiical pay spread 
among occupational groups. As shown in Appen- 
dix II, the upper and lower rounded monthly pay 
limits were about $8(X) and $2(X), with an average 
of approximately $525. Consequently, the interval 
500 * 599 was chosen as the eerier of the scale 
(rating 5), with 100-dollar steps above and below 



this value. To reduce the possibility of truncated 
ratings and allow for inflationary trends, the scale 
was extended one point on botli extremes to in* 
elude ratings of 9 and 1. The final scale, 
considered appropriate and realistic for rating of 
the 100 job samples, was as follows: 

RiUng Monthly pjy Scile 

9 S900 or more 

8 S800 to S899 

7 $700 to $799 

6 $600 to $699 

5 $500 to $599 

4 $400 to $499 

3 $300 to $399 

$200 to $299 
1 $100to$199 




10 



III. RESULTS 

The Zinalyses of interest for the three conditions 
were concerned with differences in reliability of 
ratings, distribution? of jobs, means of the job 
ratings, and mean rating time. Results are reported 
according to this sequence of analyses. 

Reliability 

Comparisons of the reliability of job pay ratings 
for the three conditions were of principle concern. 
For this purpose, inlraclass reliability coefficient 
procedures^ described by Haggard (1958) and by 
Lindquist 0953) were used, Results of these 
analyses for each condition are given in Table 2. 
To interpret these coefficients, an explanation of 
the logic and derivation of values is provided. 

Although the number of raters in each con- 
dition was held constant (AT = 150), the number of 

* with rcgird to notation, Haggard(p llj uses Rand 
Lindquist (p. ji61 ) uses T\ j to denote ir traclass correlation 
coefficients. The notation used here, ] , was adopted in 
order to maintain a distinction between intraclass cone* 
lation and symbols used to denote multiple correlation 
(R) and product-moment correlation (r). The Rj i coefh* 
cients are basic computed values inserted itito the 
Spearman Brown prophecy formula lo estimate the mean 
reliability of job ratings when the n jmber of ratings per 
job (k) is increased or decreased. The formula used to 
estimate mean reliability coefficients (R|^) for various 
sample sizes wis 

R , L-MRii) 

1 +(nlJ(R,,) 

where 

n= amount of increase or dccre^LSC in 
k for various sample siics 



ratings per job (k) obtained was different for the 
three conditions (individual k = 30; 3-man panel, 
k = 10; 5-man panel, k = 6). This design was fol- 
lowed so that reliability could be expressed in 
terms of a given number of raters per job, as well 
as a function of number of ratings per job. How- 
ever, in the determination of reliabilities for the 
three conditions, the different number of ratings 
per job was considered. This was accomplished by 
using the Spearman-Brown procedure described by 
Lindquist (1953, p. 361) and extended by Christal 

ai (1960, p.5) to estimate reliability in a job 
evaluation situation. Such an approach permitted a 
joint comparison of reliability and assessment of 
economy in terms of time and number of raters 
required. 

Following this rationale, the most meaningful 
comparisons appeared lo be among reliability 
estimates of job pay ratings which accounted for 
the different number of ratings per job, yet in- 
volved the same number of raters in each con- 
dition. This involved the computation of reliability 
coefficients (Rj^) which were estimates of the 
tneatis of k ratings per job for various size rater 
samples. To determine these values, “single-rater” 
coefficients (Ri i ) which reflect inter '‘ter consist- 
ency were first obtained, then inserted into the 
Spearman-Brown formula to estimate the reli- 
ability of mean ratings for various sample sizes 
(i.e., when number of raters was held constant but 
number of ratings per job differed for each con- 
dition). The R] 1 and n values used to determine 
number of ratings per inserted into the 
formula for delermin i for various size 

samples are included in mole 2. 



Table 2, Reliability Estimates (Rj^j) for Various Sample 
Sixes in Three Rating Conditions 



Rating 

Corsditlors 




Hy Sample sixe 




75 Si . 

tS Pcrlob^ 


15 0 Si . 

30 per Job® 


JJ5 5l . 

45 perjob® 


300 Si . 

60 per job® 


L Individual 


.9370 


.9675 


.9781 


.9835 


2. 3-man panel 


.9170 


.9567 


.9707 


9119 


3. S-man panel 


.8669 


.9287 


.9513 


.9630 



*Bask Rj 1 coefficcnls inserted into Spearman Brown formula to determine 
Rj^* [ndividual. .4978; 3 man, .6884; 5-man, .6846. 



n values (ratios) used to determine number of ratings per job in Sp 
Brown formula: 

[ndisidual, n - 1.0 (e.g., IS ratings from IS raters) 

3-man panel, n = 1/3 (e.g., 5 ratings from 1 S rateis) 

5-man panel, n * 1/S (e.g., 3 ratings from IS raters) 



earman- 
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As shown and compared in Table 2, given a 
constant number of raters and accounting for 
number of ratings per job, the mean reliability 
coefficients (R|^) were larger for the individual 
condition than for the two panel conditions, For 
example, these results indicated that 75 subjects 
divided into 3-man or 5-man panels did not give as 
reliable mean estimates as 75 individual nters, 
Further, the larger reliability estimates obtained 
by averaging across individual raters held for all 
the various size samples. In regard to the magni- 
tude of the reliability estimates, the obtained 
values compared closely to results obtained 
in other job evaluation studies (Christal et al, 
1960; Hazel & Cowan, 1966). Generally, from 10 
to 15 ratings per job yield very stable mean 
estimates. 

distribution of Jobs 

To determine whether the distributions of 
mean ratings of the 100 jobs for the three con- 
ditions were significantly different, a chi-square 
test was computed. This test was based on the 
frequency distributions of the 100 job means 
(rounded) for each condition, as shown in Table 3. 
Because the frequencies of the extreme scale levels 
(ratings 9 and 1) were zero or small, the two upper 
categories, 8 and 9, and the two lower categories, 
1 and 2, were combined. The results of this 
analysis shown in Table 3 revealed there were no 
significant differences in distributions for the three 
conditions (p > .05). 

Mean Job Ratings 

The means of the job ratings for the three con- 
ditions, including number of cases on which these 
means were based, are given in Table 4. An 
analysis of variance was performed to determine if 
there were significant differences among the three 
means. As shown in Table 4, the lesults of this 
analysis revealed no significant differences among 
means (p > .05). 

Time Required for Ratings 

To test for differences in the mean time re- 
quired to complete ratings of jobs in the three 
conditions, a multiple range lest was used 
(Edwards, 1960, p. 136). This analysis was based 
on the time in minutes for each individual (con- 
dition 1) or panel (conditions 2 and 3) to rate 20 
jobs. Induded were 150 sets of 20 job obser- 
vations for condition 1 , 50 sets of observations for 
condition 2, and 30 sets of observations foi' con- 
dition 3. Results of the analysis are given in Table 



5, which includes mean differences. As shown, 
there was no significant time difference (p >.01) 
between conditions 1 and 2, or between 2 and 3. 
However, the individual time was significantly 
smaller (p = .01) than the 5-man panel time. There 
was also a definite trend for mean time to increase 
as panel size increased. 



Table 3. Distributions of Mean Pay Ratings 
ui Three Rating Conditions 



Ratin9 


Distribution at Rating Level 
by Condition 




i_evei 

[Mean) Individual 3-man panel S*man 


9 0 


0 


0 


8 4 


7 


r 


7 8 


8 


5 


6 16 


12 


13 


5 22 


22 


22 


4 30 


27 


22 


3 14 


11 


18 


2 5 

1 1 


12 

1 


7 

1 


= 11.59; df = 


12; p >.05 




Table 4. 


Mean Pay Ratings for 




Three Conditions 




Rattng 




Mean 


Condition 


N 


Rating 


1. Individual 


3,000 


5.18 


2. 3-man panel 


1,000 


5.03 


3. 5-man panel 


600 


5.18 



F = 1.55; df^ 2/4597; p> 05 



Table 5. Mean Rating Times and Differences 
Between Means for Three Rating Conditions 



Dirrcrence Betweer\ Meant 
for Rating Contfitiont 



Rating 

Condition 


3-man 

Individual panel 


5-nian 

panel 


1. Individual 




2.23 


6.76 


2. 3 -man panel 


2.23 


- 


4.53 


3. S-man panel 


6.76 4.53 

Mean Rating Time 
for Condition* 






(1) 


(2) 


(3) 




4. 34 


6.57 


11.10 



*Mcans not iiiidcrscorcd by the same I ic arc signific antly 
different {p = .01). 
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Since lime required for ratings ani number of 
raters involved are practical matters which must 
also be considered in determining whether an in- 
dividual approach or a panel approach is prefer- 
able, the simultaneous presentation of certain 
reliabflily and time requirement resu'ts appeared 
desirable. As shown previously, the reliability 
estimates tor the individual ratings were larger 
than for the panel ratings, and there was a trend 
for mean time to increase with panel size. Such a 
joint presentation would help deomonstrate more 
clearly whether averaging across individual ratings 
or obtaining a consensus from panels yielded more 
stable job ratings, and would also permit evalua- 
tion of the .'^ficiency of the two approaches with 
regard to these practical matters. 

The mean reliability estimates and total 
number of man-hours (approximate) required to 
accomplish ratings of the 100 jobs by 150 subjects 
in each of the three conditions are given in Table 
6 Data sources and method of ascertaining man- 
hours for these values are included. 



As reflected in Table 6, the individual condition 
provided more stable estimates and was much 
more efficient in terms of a time requirement than 
the panel conditions. Since the 5-man panel 
yielded the lowest rating reliability, yet required 
2-1/2 more man-hours than the individual situa- 
tion, the use of pa^^els of this size appears rather 
inefficient. 



Table 6. Reliabflily Estimates and Man-Hours 
Required for Three Rating Conditions 





Rating 

Condition 


RalfabHIty 

Eitlmale* 


Man-hourt 

Required^ 


1. 


Individual 


.97 


10.85 


2. 


3- man panel 


.96 


16.43 


3 


5-man panel 


.93 


27.75 



values (rounded) from Table 2 for 150 5s. 



' ompuced by multiplying mean time for rating con- 
ditiun in Table 5 by number of rater (ISO Ss) and divid- 
ing by 60 (minutes). 



IV. DISCUSSION AND CONCLUSIONS 



This study investigated the problem of how to 
obtain more stable mean job ratings, given a con- 
stant number of raters. Specifically, it compared 
reliability estimalc» of job ratings determined by 
averaging across ratings obtained individually with 
reliability estimates of ratings of 3-inan and 5-man 
interacting panels. 

Intradass correlation analyses comparing job 
pay ratings revealed that estimates of niean reli- 
ability were larger for the individual condition 
than for the 3-man or 5-man panels, when esti- 
mates accounted for differir.g numbers cf ratings 
per job. This finding was consistent when pro- 
jected across samples of various sizes. Other results 
did not indicate any slatistical'y significant differ- 
ences between individual and gioup job pay ratings 
except for the time required for ratings in an 
individual i>ersus a 5-man panel situation. Th;- 
distributiorts of the mean pay ratings for the 100 
jobs were similar (p > .05), and there were no 
significant differences (p > .05) among rating 
means for the individual, 3*man panel, and 5-man 
panel conditions. 



When both reliability and time required to 
collect ratings were considered, the evidence in- 
dicated that the ir;dividual rating situation is pre- 
ferable and more economical than the use of 
panels. There was a trend fo! rating time to in- 
crease as a function of panel size. However, there 
was not a concurrent increase in reliability with an 
increase in time required for rating. This finding 
tends to agree with a previous study (Hazel, 1966) 
which found reliabiUly did not increase signifi- 
cantly as a function of rating when job evaluations 
were made individually. 

The present findings tend to support the pro- 
cedure of averaging across individual ratings rather 
than dividing raters into panels and obtaining con* 
sensas group ratings. Tut mean reliability esti- 
mates for interacting 3-man or 5-nian panels were 
not superior to the meari reliability estimates for 
individual raters. In terms of reliability, economy 
of lime, and number of raters, the averaging of 
individual ratings appears more efficient. More 
rater* and lime may be rectuired for the panel 
situation than for the individual situation to 



achieve comparable estimates of reliability. In 
addition, the number of man-hours required to 
obtain raliiigs from 5-man panels was ofsufficient 
magnitude to suggest that larger sized panels may 
be inefficient from a practical standpoint. 

Certain caution appears warranted, however, in 
generalizing results from the present rater groups 
with simple job descriptions to typical Air Force 
boards or panels, (isitally experienced sem'or 



officers comprise Air Force boards, and more 
complex and detailed job descriptions are used for 
operational job evaluations. Replication of this 
study using senior officer raters appears desirable. 
However, a study related to the question of 
generalization of results (Hazel A Cowan, 1966) 
found agreement among four rater groups with 
regard to reliability and homogeneity of job 
ratings. 
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A PPENDIX 1 . ADMINISTRATIVE PROCEDURES AND INSTRUCTIONS 
FOR EXPERIMENTAL CO' .,'ITIONS 



To obtain tlte desiied pay ratings on 100 jobs Tk n 450 basic ain-nen under three conditions, the 
following procedures and instructions were establi^ed. 

Experimental Conditions 

1. rdrmyi. The 150 subjects were required to rate job descriptions using a 9-point scale. 
Thirty ratings per job for five sets of 20 descriptions (i.e., 100 jobs) were required. 

2. panel ratings. The 150 subjects w^ere randomly divided into fifty 3-man panels on entry 
into the testing room. Each panel rated 20 job descriptions using a 9-point scale. Ten ratings per job for the 
100 descriptions were required. 

3. S-man panel ratings. The 150 subjects were randomly divided into thirty 5-man panels. Each 
panel rated 20 job descriptions usinR a 9-point scale. Six ratings per job for the 100 descriptions were 
required. 



The materials pro\ided . |] si.bj :cts consisted of a Test Record Card and list of 20 job desorptions (for 
individual or panel use). The 10( J ji b descriptions were divided into five sets of 20 descriptions, and these 
sets were the same for all three conditions. Each set contained instructions, coding information, and a 
9-point monthly pay scale. 

The Test Record Card was used to obtain certain subject identification and other personal data. All 
materials distributed to all raters had a 6-digit prenumber code airlxed to identify the experimental 
condition, job set number, panel number, and subject test number. 

Administrative Procedures 

On the day of testing, 150 basic airmen were randomly selected for each of the three experimental 
conditions. The 3-man and 5-man panels w^ere treated similarly, except for number of panel members. 
Essential procedures and instructions applied to each condition w^re as follow's. 

Condition J. /ndividuaL For this condition, all subjects were tested simultaneously. On ertry into 
(he testmg room, each subject was provided with a Test Record Card, written instructions, and a job 
descripti )n list. Instructions were also read aloud to subjects. 



The Air Force is interested in your opinions concerning various jobs. Individually, each of you 
will rate several rather familiar jobs on a 9-point pay scale to show what you consider as the 
proper monthly pay for each job. In brief, you will indicate the amount of pay per month 
which you believe (hat w’orkersin 20 jobs should receive. 

1- Fiiil. yo\i will comf^ete (he Test Record Card. Do this now . Note the testing number 
in the upper right corner of this card matches the testing number on the Job Description List. 

When you have finished the Tcit Record Card, wait until the examiner leDs you before 
you begin your ratir^gs. 

2. .Second, w'rite your name at (he top of the Job Description List . You are to rate all 20 
jobs ming a number from 1 to 9 which is indicated on the rating scale at the top of your Job 
Description List. For instance, if you believe z job on your list should be paid from S500 to 



Materials 



INDIVIDUAL INSTRUCTIONS 
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$599 per month, write in a 5 in the left-hand space opposite that job. Consider monthly pay 
based on a normal work week for each job. The jobs you rate may be professional, salaried, 
wage, or self-employed, and may be paid weekly, bi-wcekly, monthly or on an hourly or 
piece-rate basis. We are interested in your best estimate of monthly pay so there are no right or 
WTOng ratings in this task. Hold your hand up if you have any questions. 

3. You are now ready to begin your ratings of the 20 jobs. At the top of the Job 
Description List enter the time you start your ratings. 

4. After completion, wiite in the time you completed your ratings. Put the total amount 
of time in minutes you took to make your ratings in the appropriate bo.K. This is not a timed 
activity and you will be given all of the time you need. We want to know how long it lakes you 
to make the ratings. 

5. "^ The Examiner will excuse you from the testing room when you have completed your 
task. Leav'e all of your material on your desk, 

READY : Commence your ratings. 

Conditions 2 and 3. 3-man panel and 5-tnan panel. On entry into testing room, the 150 subjects for 
each of these two conditions were randomly divided into fifty 3-man panels or into thirty 5-man parcels, as 
appropiiate. Large testing rooms were prepared in advance so there was wide spacing between panels, When 
the gioups were assembled, the following material was distributed to each panel: set of instructions (for 
3-man panel or 5-man panel); Test Record Card for each subject; one “Panel Report” job description list 
(on a random baas to each panel); and lists of job descriptions for use by each panel member. 

The instructions, as appropriate and separately for the two panel conditions, were read aloud to all 
panel members. 



PANEL INSTRUCTIONS 

The Air Force is interested in your opinions concerning various jobs. You will be asked *o rale 
several rather familiar jobs on a 9-point pay scale to show the proper monthly pay for each job. 
In brief, you will indicate the amount of pay per month which you believe that workers in 20 
jobs should receive. You will do this as a 3-nian (or 5-man) group or pai.el. 

L First , look at the package of material that has been given your 3unan (or 5-m.''n) 
panel. Lay the “Panel Report” sheets and Job Description Lists aside and let each panel 
member lake one of the Test Record Cards. 

Each panel member will complete a Test Record Card, Do this now . Note the testing 
number in the upper right corner of this card matches the testing number o(i your Job 
Description List. 

When you have finished the Test Record Card, wail until the examiner gives you further 
instructions befoiC you begin your ratings. 

2. Second , as a panel you are to determine a single rating for each of the 20 jobs. As a 
group, you are to rale the 20 jobs using a number from 1 (o9 which is indicated on the rating 
scale at the lop of your Job Description List. For instance, if you believe a job on your list 
should be paid from S500 to $599 per month, write in a 5 in the left-hand space opposite that 
job. Consider monthly pay based on a normal work week for each job. The jobs you rate may 
be professional, salaried, wage, or self-employed, and may be paid weekly, bi weekly, monthly 
or on a hourly or piece-rate basis. We are interested in your ^&i estimate of monthly pay so 
there arc no right or wrong ratings in this task. 

Each panel can decide its own method of operation. Use the “Panel Report” to record 
these ratings. Each p?nel will rate all the jobs using the ] (o 9 scale at the top of the Panel 
Report and enter their ratings in the left-hand spaces opposite each job. 
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The panel judgment should be arrived at after discussing and considering all factors 
applicable to the job. Each panel will work independently and not disturb other groups. As a 
panel you should aD agree unanimously that the value you record is your panel’s best judgment 
of the appropriate monthly pay for the job. Each member of the panel has an opportunity to 
voice his opinion. 

On the “Panel Report” carefully record the starting time, time of completion, and 
number of minutes needed to make your panel ratings. This is net a timed activity and each 
group will be given all the time needed. However, we are inie^ested in knowing the correct 
amount of time it takes you to make the ratings. 

3. The examiner will excusr each panel from the testing room when each group has 
completed its task. Leave all material on your desk when completed, 

READY : Commence your ratings. 
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APPENDIX II. MEDIAN ANNUAL AND MONTHLY INCOME 





OF MALE CIVILIANS FOR MAJOR 
OCCUPATIONAL GROUPS 






Occupational Group* 


Annual 

Income" 


Monthly 

Income 


1 . 


Professional and technical 


59,205 


$767 


2. 


Farmers and farm managers 


3,547 


296 


3. 


Managers, officials, and proprietors 


8,826 


736 


4. 


Oerical 


6,542 


545 


5. 


Sales 


7,553 


629 


6. 


Craftsmen and foiernen 


7,161 


597 


7. 


Operatives 


6,135 


511 


8. 

9. 


Private household‘s 

Service (except private household) 


5,1 17 


426 


10. 


Farm laborers and foremen 


2,576 


215 


n. 


Laborers (except farm and mine) 


5,133 


428 


12. 


Occupations not reported^ 







*From Reiss 0961), pp. 154 155. 

^ f^Tom Statistfcat Abstract of the Vntted States ( 1968), Table 335 present- 
ing data for 1966. 



^Omitted in Statistical Abstract. 
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APPBNhIX HI. FIVE SETS OF JOB DESCRIPTIONS 



Set 1 



Keu>s Reporter. Collecls perlinenl fads by interview or investigation about events having news value. 
Covers special assignments. Writes stories for newspaper publication. 

Purchasing Agent, Retail Store. Purchases materials and supplies at most favorable price, necessary for 
operation of store. Approves spcciOcations, draws up contracts, and approves bills for payment. 
Keeps records of purchases. 

Automobile Washer. Cleans inside with broom, doth, mop, and vacuum cleaner. Cleans outside with 
soap, water, sponge, and chamois. May wax and polish the outside. 

Phannacist Compounds and dispenses medicines and preparations prescribed by licensed physicians. 
Perfoni.-s tests to determine identity, purity, and strength of drugs. Maintains stock of drugs and 
cliemi cals. 

.Airplane Mechanic. Overhauls, services, and inspects airplanes, except engines, both on the flight line 
and in the shop. Repairs and replaces such parts as fuselage, control cables, tanks, gears, and lines. 

Hospital Orderly. Assists nursing staff in hospital. Lifts patients in and out of bed, carries meal trays, 
and cleans rooms, ^ - ■ 

Criminal Lawyer. Specializes in law cases dealing with crime such as murder and theft. Defends client 
on charges made against him, examines and cross-examines witnesses, summarizes case and makes 
closing remarks to jury. 

Athletic Instructor. Gives individual or group beginning or advanced instruction in calisthenics, 
gymnastics, and other exercises. Teaches use of various athletic equipment. Organizes, instructs, leads, 
and referees such games as baseball, football , basketball, etc. 

Employment Manager. Interviews applicants, hires or refers those with satisfactory qualifications to 
proper department. Adjusts disputes and grievances. Reviews transfers and discharges. Conducts 
research on wages, hours, and working conditions. 

Commercial Airline Pilot. Operates an airplane for transportation of mail, passengers, freight, or other 
commercial purposes. Operates airplane on scheduled or chartered flights for an airline. 

Shipping Clerk. Prepares merchandise for shipment oi delivery. Prepares bill of lading, sorts articles, 
and keeps records of articles shipped. May receive incoming goods. 

Tailor. Makes tailored gaiments such as suits, topcoats, and dress clothes. Designs, measures, i^ws 
(hand and machine), and fits articles as required. May supervise other workers performing similar 
tasks. 

Automobile Mechanic. Repairs passenger automobiles and light trucks. Disassembles and assembles 
and overhauls engines, transn\issions, rear ends, and other assemblies. 

Rookkeeper. Keeps systematic set of records of business transactions of an establishment. Balances 
books and compiles '■eports at regular intervals to show receipts, expenditures, accounts payable and 
receivable, and profit or loss. 

Librarian Assistant. Assists librarian by cataloging books, replacing books on shelf, and issuing and 
receiving books. Sends out overdue notices when necessary. 

Telegraph Operator. Operates telegraphic equipment for transmitting and receiving signals or 
messages. Makes adjustments to instruments Assists iv other office duties. 

Sales Clerk, P tail Receives cash from customers, \sraps articles purchased, and makes change. 
Arranges nKrehandise on shelves in a neat and orderly manner. 

Television Repdir)7uin. Repairs and adjusts television receivers, Follows schematic diagram to locate 
defects. Installs TV sets and antennas, 
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Longshoreman Loads and unloads ships’ cargoes. Moves articles on hand trucks and stacks on wharf 
or in ship. Attaches slings to heavy objects for Lfling. 

20. Nurse, Registered. Does general nurse work in homes, hospitals, and institutio.ns. Administers drugs, 
medicines, and ointments. Changes dressings on wounds and bathes patients. Prepares operating 
room, sterilizes instruments, and gives injections. 



1. President, College or University. Administrative head ot a college or university. Concerned with the 
financial and business affairs and academic life of an educational institution, 

2. Musician, Instrumental Plays one or more musical instruments in a symphony orchestra, band, or 
similar musical organization. 

3. Dental Assistant. Prepares patients for examinations, treatments, or dental surgery. Assists dentist by 
handing him necessary implements. May sterilize instruments and assist in making dentures. 

4. Draftsman Prepares accurate charts, layouts, and detailed drawings according to specified 
dimensions. Makes finished designs from sketches. Uses manual skill in manipulation of drafting tools 
and equipment. 

5. W'elder. Fuses metal parts together by means of acetylene torch or electric welding apparatus. 
Fabricates metal shapes and repairs metal objects. Cuts metal with torch burner. 

6. Physician, Genera! Practitiorer. Attends to all kinds of medical cases. Examines and diagnoses 
condition of patient. Prescribes treatment and performs minor operations. Writes prescriptions and 
administers drugs. 

7. Developer, Photographic. Treats exposed film with a series of chemical and water baths to bring out 
image. Prepares chemical solutions according to specific formulas. Dries negatives. 

8. Power SerWee Operator. Operates and maintains power-production equipment such as generators, 
meters, and high-voltage switching and control equipment. Operates pumps, compressors, or blowers. 

9. Bank Cashier. Acts as executive officer of bank. Approves loans, collects debts due bank, and buys 
and sells collateral. Manages bank and supervises all subordinate officers. 

10. fAectrical Engineer. An engineer who plans and supervises construction and operation of 
el cc trie-power systems, communication facilities, and electronic equipment. May also engage in 
research, inspection, and consultation concerning complex electrical equipment. 

11. Watchman. Guards an industrial plant, warehouse, or other property against fire, theft, or illegal 
entry. Makes periodic inspection tours of building and grounds. 

12. Typist. Typewrites letters and addresses envelopes. Fills in reports and forms. Types copy from rough 
draft or .orrecied copy. Makes stencils for reproductions. 

13. Dry Cleaner Operator. Operates dry cleaning michine to clean garments, drapes, and other articles. 
Determines proper chemicals to use and filters cleaning chemicals after use. 

14. Teacher, High School. Teaches pupils in church, private, or public high schools (secondary schools) in 
one or several subjects. May perform duties as student advisor. 

15. Brkkmason. Lays brick, tile, and similar building blocks to construct various structures. Ch<^cks 
alignment of structures. Performs other duties related to bricklaying. 

16. Firefighter (Fireman), Rides on truck to scene of fire. Connects and nrans hose or uses portable fire 
extiiiguishi/ to pul wate. or chemicals on fire. Enters burning buildings to combat fires, Rescues 
persons overcome by fire or smoke. Administers artificial respiration. 

17. Office Machine Senkeman. Inspects, adjusts, repairs, and services office machines. May insttucl 
machine operator in correct operating procedures. 
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18. Janitor. Keeps public parts of building in clean and orderly condition. Mops and polishes floors. 
Removes litter and waste paper. 

19. Carpenter Foreman Supervises a group of carpenters on construction projects, hferprcls blueprints, 
sketches, or verbal orders. Assigns duties and inspects work. Keeps time and production records. 

20. Posting-Machine Operator, Records numerical details of transaction using various bookkeeping 
machines. Places forms in niachine and performs necessary calculations. 



1 . 

2 . 

3. 

4. 

5. 

6 . 

7. 

8 . 
9. 

10 . 

11 . 

12 . 

IJ. 

14. 

15. 

16 . 
17. 



Set 3 

Motion Picture Projectionist. Operates motion picture projection and sound producing equipment. 
Ins^)ccts, repairs, and rewinds film. Cleans, repairs, and inspects equipment. 

Electriciatx Lays out, assembles, installs, and tests electrical fixtures and wiring, Plans proposed 
installation from blueprints. Installs and checks wiring systems and electrical equipment. 

Sheer Metal Worker. Fabricates, assembles, alters, repairs, and installs sheet metal articles and 
equipment. Cuts, drills, grinds, and files metal according to blueprints. 

Machinist. Constructs and repairs all kinds of metal parts, tools, and machines. Reads blueprints and 
written specifications, working to close tolerances. 

Shoe Repairryian. Resoles, reheels, and repairs shoes- Attaches toe and heel cleats, stretches, shines, 
and replaces buttons or ornaments. 

Dispatcher, Motor Vehicle. Assigns vehicles and drivers for transport of freight or passengers. Keeps 
records of time and date of delivery, mileage traveled, and drivers assigned. 

Medical Technician. Performs medical duties in hospital or laboratory. Make tests of urine, blood, and 
infections. Prepares vaccines, types blood, and gives biological skin tests. 

Tool Maker. Specializes in the construction, repair, and maintenance of machlnC'shop tools, jigs, 
fixtures, and instruments. Operates various machine tools and performs oilier highly skilled work. 

Clergyman Prepares and delivers sermons. Performs ordinances of church. Officiates at meetings and 
supervises religious education program. Performs additional church duties such as visiting the sick, 
and giving personal advice and counsel. 

5choc>/ Superintendent. Formulates plans and policies for the administration of a city or county 
school system. Supervises the appointment, training, a d promotion of teachers and principals 

Bootblack. Cleans and polishes leather, si-edc, canvas, rnd other types of shoes. Assists patrons w ith 
coats and brushes coats and hats. 

Cook. Prepares, seasons, and cooks soups, meals, vegetables, and dcsserls. Usually specializes in 
specific food dishes. 

3/eat Cutter. Cuts, trims, and bones meal using knives, sa'vs, or oll:er instruments. Chops and grinds 
meals using hand cr powered equipment. Prepares various nieal products. 

Power Shoivl Operator. Operates Diesel or gasoline powered equipment to excavate or mow material. 
Manipulates levers and pedals to move machine and lift boom or shovel. 

Secretary. Performs general office work and clerical duties. Takes and transcribes dictation, makes 
appointments for executives, answers and makes plione calls. Handles mail and writes routine 
correspondence. 

Chemist. Performs analytical and research work in the general field of chemistry. Makes chen.ical 
tests on foods, drugs, dyes, paints, petroleum products, etc. Develops new processes. 

Optometrist. Examines eyes and prescribes treatment to improw vision without use of medicine, 
drugs, or surgery. Refers patients with eye diseases to medical practitioner. Tests, prescribes, and fils 
lenses and glasses to correct visual deficiencies. 
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18. Civil Enghwe/. An engineer who plans, designs, and supervises the construction and maintenance of a 
large variety of structures or facilities such as roads, bridges, airports, power plants, water and sewage 
systenrs. 

19. Chemical EngiNcer. Ad engineer who applies chemistry and various branches of the engineering 
sciences to the design, construction, and op: ration of equipment for carrying out chemical processes. 
Conducts research to develop new and improved chemical manufacturing processes. 

20. County Judge. Piesides as an arbitrator, advisor, and administrator of justice in a court of hw. Deals 
with proving of wills and the administration of estates, checks accounts of estate executors, 
establishes rules of procedure for his court. 



1 . Mailtikiti. Delivers mail to private homes or businesses along an established route after sorting the mail 
according to streets and street numbers. 

2. Telephone Operator. Operates telephone switchboard to relay incoming calls and make connection 
with outside lines by using a system of cords and switches to plug in correct circuit. 

3. Surveyor. Supervises and directs w^ork parties in determining locations and measurements of points, 
elevations, and contours of earth’s surface. Calculates information needed to make maps, charts, land 
valuations and deeds. Verifies accuracy of survey data. 

4. Coal Miner. Drives mine shafts, sets explosives, cleans debris, installs timbering and supports, shovels 
coal into mine cars. May operate machine to undercut coal and load into cars. 

5. Barber. Cuts hair according to instructions of patron. Trims neckline using razor. Gives shampoos, 
scalp treatments, and shaves. 

6. Painter. Paints sheds, houses, and other buildings. Mixes paint and matches colors. Erects scaffold. 

7. Cashier. Receives and examines incoming cash, coi’nts rnoii^/, makes change, and prepares bank 
deposits. Prepares payroll and pay checks. Keeps record of cash transactions. 

8. Aeronautical Engineer. An engineer who specializes in the design, construction, or testing of aircraft. 
Analyzes stresses to deterrrh^e if aircraft will function properly under fli^vht conditions. Supervises 
assembly of eqioipment and technical phases of air transportation. 

9. Architect. Plans, designs, and oversees construction of buildings such as private residences, office 
buildings, factories, and similar structures. Furnishes professional ad\ice on such matters as cost, 
design, materials, and equipment. Plans layout and prepares sketches of proposed buildings. 

10. Electrician Poretnan. Supervises a group of electricians. Interprets blueprints, sketclies, or written 
orders. Deterntines work procedure, A^sigiis duties and inspects quantity and quality of work. Keeps 
time and production records. 

1 1. Librarian (I ibrary Manager). Manages a library. Supervises assistants and perforirs duties according to 
size of library. Selects books to be purchased. Supervises cataloging, ciassiftcalion, and circulation of 
books and periodicals, 

12. Airplatic Savigator. Locates position and directs course of airplane in ijight through the use of 
navigational instruments and charts. Directs changes in course due to weather conditions. Keeps log 

of night- 

13. /fducufioDdi Piychologisi. Investigates processes of ntcnlal growth and dcvelopn>ent io guide 
individuals in the selection cf academic or vocational courses leading to a suitable career. Analyzes 
causes of n^ladjustment of individuals in schools and recommends corrective action. 

14. Dentist. A person engaged jn any phase of dentistry such as extracting, cleaning, filling, or replacing 
teeth; performing corrective work such as straightening teeth; treating diseased tissues of the gum; 
performingsurgical operations on jaw or mouth; and making and fitting false teeth, 
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Bus Driver. Transports passengers according to deflriite time schedule. Collects tickets or fares from 
passengers and keeps records of receipts. Assists passengers with baggage. 

Traffic Manager, Air Transportation. Establislies fre'ght and passenger rates in accordance witli 
company policy. Consults and makes suggestions on problems related to traffic movement. Supervises 
Operation of reservation and ticket offices. 

Baker. Produces finished baked goods such as bread, cake, and pastries. Measures ingredients, mixes 
dough, balces products in oven regulating timing and temperature. 

Plumber. Assembles and installs air, gas, water, and waste-disposJ svstems. Tests Joints and pipe 
system for leaks. Installs gas, water, and sanitary fixtures. 

Watch Repairman. Cleans, adjusts, repairs, and oils watches and clocks. Checks for magnetism and 
demagnetizes watches when required. Performs jewelry repair. 

Aircraft Engitic Mechanic. Inspects and overhauls airplane engines. Examines engine operation for 
malfunction. Performs additional duties such as flushing crankcase and oiling iuoving parts. 



Waiter. Serves food to patrons Sets tables with clean linen and silverware. Removes dirty dishes to 
kitchen and cleans equipment. 

Truck Dri.vr, Heavy. Drives a heav>^ truck to deliver or transport merchandise. Usually loads and 
unloads truck. May make minor repairs to vehicle. 

Laborer (Construction). Erects, repairs, and wtecks buildings, bridges, and other types of 
construction. Digs ditches, mixes concrete, and uses various hand tools. 

Corporation Lawyer. Advises corporation concerning its legal rights, and the advisability of 
prosecuting or defending a law suit. Acts as an agent of. the corporation, and seeks to keep 
corporation from expensive lawsuits. 

Personnel Manager. Formulates policies relating to the compensation, trairiing, promotion, and 
w'eJfare of employees. Supervises subordirrates in carrying out these policies. 

Radio Operator. Controls the operation and adjustment of all transmitters and receivers in a radio 
station. Makes minor repairs Sends and receives radioteletype messages. 

Veterinarian. Studies and treats diseases of animalsN Advises on care and breeding of animals. Ins^iccts 
animals intended for human consumption. 

Radio Repairman. Tests and repairs defective radios. Test circuits aiul tubes using various testing 
meters and devices, Installs radios and antennas. 

Matiager, Retail Automotive SerxHce. Manages a retail automobile service station which services autos 
witih gas and oil. Supervises ejnployees. Makes reports, takes inventory, and ensures premises are 
clean. 

Certified Public Accountent. An accountant who has passed a slate certification examination. 
Performs a variety of accounting services. Certifies financial statements, conducts financial 
invcstigatior.s, and prepares or reviews tax returns. 

Messenger. Sorts and delivers letters, messages, packages, documents, and records to various offices 
and sections within an establishment. Keepi records and re,:eipts of articles delivered and received. 

Detective (l*olice). Assists uniformed police in prevention of crime. Investigates known or suspected 
criminals. Examines scene of crime and questions individuals or witnesses concerned. Keeps records 
of such information and reports to suforior officer. Arrests crinvnals when guilt has been established. 

Physicist. Conducts research into phtses of physical phenomena such as motion, gravity, laws of 
liquid pressure, heat, Light, and eleclroriics. May monitor scientific projects and serve as consultant. 
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14. Professor, College or Utiwerstty. A recognized authority in a particular field ofknowledge. Conducts 
graduate and undergraduate classes in his field and oversees the research of others. Conducts research 
in his own a:ea of interest. 

15. Personal Services Director. Directs welfare actinties for employees of industrial and commercial 
establishments. Arranges operation of libraries, recreational facilities, and educational courses, May 
assist employees in solution of personal problems. 

16. Floor Manager, Retail Store. Supervises employees in a designated section of retail store. Instructs 
new workers, regulates hours, adjusts claims, and answers custonrer questions relative to merchandise. 

17. Carpenter. Cuts, fits, and erects the wooden framework, partitions, subflooring, and other parts of a 
building. May install trim and finish work. 

18. Lfncmart Hangs telef^one or telegraph wires on poles, cuts in feeder lines, and attaches appliances 
for telephone or telegraph communication. 

19. Calculator Operu or. Operates automatic calculating machine to check such records as inventories, 
fin-ncial accounts, pay rolls, invoices, and statistical reports, 

?.0. Garage Foreman. Assigns duties to mechanics and inspects their work for qu*'.lity and quantity. Keeps 
time, production, and other clerical records. May hire and discharge workers. 
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